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(57) ABSTRACT 

Methods and associated structures and systems for automat- 
ing software test procedures so as to enable automated black 
box and white box testing techniques to be performed in an 
automated manner. The present invention further provides 
for test sequences to be easily maintained in synchronization 
with corresponding changes in the underlying source code 
files. In particular, white box and black box test sequences 
are described in a test language of the present invention in 
a manner which is embedded within comments of the 
programming language used to implement the software 
product. Such comments are ignored by software develop- 
ment tool setSrTesting -tools of the presentlnvention-parse 
^the^sourcexode-filesto'ex 

(sequencesZdefined_in_the_comments thereinTThe test 
sequences so extracted are then performed by the test tool in 
cooperation with a software debugging tool for purposes of 
applying stimuli both for black box testing of the software 
product as a whole and for white box testing of internal 
functions and modules of the software product. 
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METHODS AND SYSTEMS FOR iniernal structure of the software product. Such interna! 

AUTOMATED SOFTWARE TESTING structure implements the requisite function of the product 

but is otherwise substantially transparent to the user of the 

BACKGROUND OF THE INVENTION software product. Rather, the user of the software product 

5 applies inputs to the software product in accordance with the 

1. Field of the Invention specifications of the program and expects resultant outputs 
The present invention relates to software testing tech- in accordance with the specifications. 

niques and in particular to methods and associated systems Following initial development of a software product, an 

and structures for automated software testing which inte- iterative process follows in which the software product is 

grates test definitions with source code of the software lo tested against specifications of its intended function, Prob- 

product under test. lems found by such testing procedures are corrected by the 

2. Discussion of Related Art programmer(s) by locating the reason for the problem using 

, / 1 f J * u • debugging tools and techniques, editing the source code files 
Computer programs (also referred to herem as software a -v .u jt-ii • u^t. 
. , V I . ^ . . u- u • % re-compUmg the source code. Following such debue- 
products) evolve m an cngmeermg process which is itera- . ■ v, • . . ^ r 
f. A Ji * -i- u 1^ gmg sessions, the program is re-tested verify proper opera- 
tive. A soitware development process involving software ? ° i « T . , . ; • . e 

design engineers is used to initially design and implement ^°°h ,.1 n. " 1,1 T h h T "'"^ °^ 

the software product. Testing of the iiitial-design then .^'""^^f^^^"* "^T^ """^ '^^'^ ''^''^ T'' ^ 

verifies operation of the software product.(Oftea.theTe^ndn "l °,k f P^'g^'"'"," °f "y teams often 

development-proceSeTSF^iraTTtmH toj ^ Jnrin^ers development engineers and 

complete thedevelopment of the software^pfoduct-tb-a-point-?" , , . ., 

^.hcre, the program is^iSBstantiaUy free of errors (also J , 8*?" i m^y^^ data 

tefcricd'lo-as bugs). ' --—J (generally referred to as stimuli) as input to the software 

, , J . . • ■ product and determining that the program's behavior and 

In general the program development process "evolves a ^ ^ accordance with functional require- 

programmer designing a solution for a problem descnbed by ^^„j3 specifications. When the testing engineer or 

a specification or functional requirement. The programmer ^ ^.^^^^^ ^ 

proceeds first at a high-level envisiomng abstract modules development engineer or development team, it is common 

and functions to be performed. The high level abstrac ions ^^^^ ^ ^^^.j , „ft^„ ^.^^^ t„ 

are then further decomposed by the programmer (also ^ox" testing. Such testing is referred to as black 

referred to as software engineer) mto corresponding lower ^ox in the sense that daU or stimuU is appUed as input and 

levels in which the higher level design is miplemented in a ^ ^^^Xn^^tA without inspection, or even neces- 

computer pro^amming language. For example, program- ^^jj kno^^dge of, the internal structure of the software 

mers often utilize the C programmmg hmguage. the C++ ^^^j ^^^^^^ ^ ^^^^^ ,^ ^^^^^ ^^^^ 

language, or the Java programmmg language for implement- ^ j^^^^ ^j^^^ ^g^^^ ^^^^ ^ ^ y^^^ ^ 

mg a particular software product. Components of the soft- ^^^^^^ ^„ j ^^^^^^ ^^^^-^^ ^1^^^^, 

ware product written m such programming languages are ^ ^^^^^ j^^j ^3^, ^^^^^^^^^^ 

often referred to as source code. The source code IS created ^ i ^ i i.- l .i- • . i 

, 11*- rni Convetsely, testmg procedures which utilize internal 

by programmers and stored m collections of files often , , , , j . a ^ - - 

c J , J r:i knowledge of the software product structure and function is 

referred to as source code files. r j , « u . u * * u% u 

often referred to as white box^ testing. The term white box 

These programming languages are not directly executed testing refers to the fact that the tester has knowledge of the 

by computers. Rather, computers execute instructions at a -^^^^^^^^ ^j^^ture and design of the software product under 

much lower level often referred to as machme code. The test. For example, the testing engineer may direct particular 

programs wruten m source code programming languages are ^^^^^1^ data as input to the software product 

then compiled, a step involving the translation of the source knowing that the internal structure may be better exercised 

code files from the selected computer programming Ian- thereby. Or, for example, white box testing may involve 

guage to a machine code format suitable for execution by a j^sign and application of a test sequence directed to exer- 

general-purpose computer. ^^i^ing particular fiinctions within the software product. 

Programmers use a wide variety of sophisticated tools in Tools for automating test procedures have evolved largely 
performing software development tasks. Such software independent of the tools for programmers to design and 
development tools often tightly integrate a variety of indi- 50 debug software products. For example, it is known in the art 
vidual components utilized by software engineers in the to create and utiUze automated testing tools which permit the 
development process. For example a text editor is often a test engineer to configure a sequence of input stimuli to be 
first component utilized by software engineer to construct applied to the software product and to automate the gather- 
source code files for a particular product. In an integrated jng of resultant output data. Testing tools as presently known 
tool set. such a text editor may be particularly customized 55 also provide the abiUty to automate the evaluation of the 
for the source programming language utilized by the pn^- generated output to determine whether it complies with 
grammer. In addition, higher level design tools and docu- requirement specifications. Operation of computer programs 
mentation generation tools may be integrated with the text are significantly dependent upon parameters outside of the 
editing process so as to document components of the even- specific input stimuli. Such external environmental condi- 
tual product design. Further tools include, compilers for tions might, for example, include the state of the computer 
translating source code into lower-level machine code oper- system on which the program runs. Testing tools have 
able on a particular computing environment. Still other tools therefore included the abUity to set up an external computing 
include debugging tools used by the programmer to test the environment in preparation for execution over particular 
functionality of the designed software product while view- test. In other words environmental parameters of the com- 
ing the corresponding source code. 55 p^ting system on which the program is run, as distinct from 

The program's source code therefore reveals it's structure input stimuli applied to the software product program, may 

and function in that the design of the source code represents be configured by the automated test tool prior to performing 
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a particular automated lest sequence on the software prod- In view of the above discussion, it is evident that a need 

uct. For example, particular configuration parameters of the exists for improved software testing tools which permit both 

software product may be defined (e.g., in a configuration black box and white box testing techniques to be applied to 

file) prior to performing a particular automated to sequence. software products in an automated fashion and in a manner 

Or, for example, configuration parameters associated with 5 which permits synchronization of the test sequences and 

the computing system in which the software product oper- changes made to corresponding software modules, 
ates may be configured prior to performing a particular 

automated to sequence. SUMMARY OF THE INVENTION 

In addition, present automated software test tools have ^, • • . . ^ » . 

included the abUity to define test sequences is in a portable The present invention solves the above and other 

manner largely independent of the design methodology and problems, thereby advancing the state of the useful arts, by 

tools used for program development. For example, present providing methods and systems for automating white box 

automated software test tools often use a script language to and black box testing techniques as applied to software 

define input stimuli and environments and to operate pro- products. Further, the present invention provides for such 

grams to analyze the generated output. A test sequence and automated testing while enabling synchronization of the test 

expected outputs defined in such a script language is often sequences with changes made to corresponding software 

referred to as a test script. Such a script language is modules. 

independent of the particular computer programming Ian- particular, the present invention provides for defining^ 

guage used to implement the software product. The syntax ^^^^ sequences as- embedded within-the source-code:3f"r 

and senaantic of the script language is usually defined by the r^^^^^^ prDduc^TTb^lT^f Ihl^^^i^iEv-^i^E^r^c- 1^ 

test tool Itself and therefore parsed and otherwise processed ^o^.oftware'-^SIIFS^ code files to extract embedded test 

by the test tool to perform desired test sequences denned m , , r ^ , * j * . 

, , ■ . . . . J ... 1 .1. r sequences and to periorm those extracted test sequences on 

the test scripts. Present automated test tools are therefore j * i_ jj j . . • 1 j ^ 

portable in that the test sequences are portable across a wide Uie software product Ihc embedded test sequences mclude 

variety of computing envkonments and are independent of definitionsof black box test sequences as well as a whi e box 

the computer programming language used to implement the 25 ^^'^ sequences. The test sequences extracted from software 

program modules are applied to the software program through coop- 

In general, it is considered desirable for a software erative use of a software debugger tool. Software debugging 
product to be tested in a manner which ensures significant tools as presenUy practiced m the art permit the testing tools 
coverage of the various execution paths within the software of the present invention to apply white box test sequences to 
product. Most computer programs include significant 3Q particular modules and functions within the software prod- 
sequences of logical controls (Boolean tests) which branch uct. The white box test sequence embedded within the 
among various paths of the software product in response to software source code modules of the software product arc 
results of tests within the program. One metric of the efficacy therefore extracted by the testing tool of the present inven- 
of software testing tools and techniques is the degree to tion and applied to the software product in co-operation with 
which all possible branches in a program are tested by the 35 the debugging tool. The debugging tool reUiras values 
test tool. generated by particular software functions for further analy- 

Automated test tools as presently known in the art are ^is by the testing tool to verify proper operation of the 

limited in their test capabilities, in large part, because of the particular function. 

limitations of "black box" testing. Though present test The white box and black box test sequences are embedded 

techniques provide significant automation for the test cngi- 40 within functions and modules of the source wde files:of:t^e> 

neer in applying stimuli and gathering output results, they ^^software producTin a manner^tr^spareWto the^ 

are limited in that through black box testing techniques is ^prograrn^mmgj^ 

difficult to effectuate testing of the internal structures and gram. For example, in the preferred embodiment, thVtest 

functions of the software product. Such black box testing sequences are embedded-within source code files-of the^ 

tools merely apply stimuli within a prescribed range of 45^software fprdd as^omments_within the particular pro^ j 

appropriate input values and verify the resultant output. ^ gr amming Janguagej uscd to impleinentlhe"program. Such 

These tools arc therefore largely precluded from effectively coninients are ignored, in general, by software development 

testing internal data structures and internal functions which tools utilized by program development engineers. Such 

are not easily accessible through application of external comments therefore serve only to document the structure 

stimuli. It is therefore difficult for black box testing tech- 50 and function of the software product and, in accordance with 

niques to assure test coverage which exercises all, or even a present invention, provide test sequences for extraction and 

significant portion in some cases, of the possible branch evaluation by the testing tool of the present invention, 

paths through the software product. Rather, it is preferable to The test sequences defined as comments within the func- 

use "white box" test techniques to assure adequate test tions and modules of the software product are therefore 

coverage of the internal structure and function of a program, 55 maintainable in a manner which permits synchronization 

It is therefore problem in software testing to provide tools with changes made to the underlying software product. For 

which automate software testing procedures in a manner example, the program development engineer may define 

which permits white box testing techniques to be applied to black box and white box test sequences appropriate for each 

software product. module or function of the software product and embed such 

Is a hirther problem in the testing of software products to 60 sequence in comments of the source code files. As the 

maintain white box test sequences in synchronization with engineer changes a particular function or module, the cor- 

changes made to the software product. Since software devel- responding test sequences may be easily modified to corre- 

opment tools and testing tools tend to be largely spond to the changes made in the underlying software 

independent, is a problem for a software development engi- module or function. 

neering teams and software test teams to maintain synchro- 65 As noted above, the testing tool of the present invention 

nized versions of test sequences and corresponding software performs embedded test sequences through cooperative 

versions. execution of a debugger tool. The debugger tool enables 
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application of black box stimuli to the program by executing locating embedded test sequence descriptions 119 within the 
the program, through the debugger tool, with particular input program source code files 118. As described further herein, 
values defined by the black box test sequence. More the embedded test sequences are preferably embedded 
importantly, the use of a debugger tool enables the applica- within comments of the programming language used for the 
tioD of white box test sequences by invoking particular 5 program source code files 118, For example, in Jhe C/C++ 
intemal functions with particular parameter values and by programming languages,':ali:text included-within any^nci- 
aUowing setting of desired values in internal data structures ment of the.progamming laigguage is generallyjgnored.for 
of the program under test. C^purposes orconapihng an^d^perati^^ test7 
, . , , M.I- sequence directives "arid parameters associated with auto- 
Smce debugger too s are widely available m most pro- moated tcstJooriOO may therefore be-embedded within such 
gram development toolsets, use of the debugger tool retams lo ^^^^^ ^^y^^ withour impacting c"o^iM55riHd"^^^ 
the portabihty features of present test tools while cnablmg r^Qjj (jj^ program. ~" — 
the automation of both black box and white box test '—Tesr^uelTces so extracted by test sequence extractor 106 
sequences. Further, smce the test sequences arc defined as are provided to test manager 108 for execution of the test 
comments embedded within the source code files of the sequences described therein and analysis of the results of the 
program, the test sequences are easily maintained in syn- jgg^ sequence execution. In particular, test manager 108 
chronization with changes in the underlying, internal struc- includes interpreter 110 for determining the test sequences to 
tures and functions. be performed from the test command and directives 
The above, and other features, aspects and advantages of extracted from source code files 118 by test extractor 106. 
the present invention will become apparent from the fol- Interpreter 110 parses the extracted test directives and corn- 
lowing descriptions and attached drawings. ^^ands to generate appropriate environments in which the 

test sequence is to operate, to invoke the program as required 

BRIEF DESCRIPTION OF THE DRAWINGS ^ perform the test sequence, and to evaluate the generated 

output and side effects of operating the program in accor- 

HG. 1 is a block diagram of a system for automated dance with the test sequence. As discussed herein below the 

testing of a program under test in accordance with the 25 interpretation of the commands to set up the initial 

present mvention. environment, perform the test, and evaluate the generated 

FIG. 2 is a flow chart describing a method of the present output, varies depending upon whether the test sequence is 

invention generalized for both black box and white box test * black box test sequence or white box test sequence, 

techniques in accordance with the present invention. 0°^® ^^^t sequence commands and directives have 

fl U.J -u- J r*u * been interpreted by interpreter 110, executive 112 within test 

FIG. 3 :s a flow chart describing a method of the present • i j . . -.^.i 

„ J . J r u 1 1 u * * manager 108 mvokes the program imder test 104 so as to 

mvention as specifically adapted for black box testing in c .uj j.. * .jl 

, .f. ,u * ' perform the desired test sequence. As noted above, test 

accordance with the present mvention. ^ • . . i_ • i- L 

^ manager 108 through interpreter 110 has initialized the 

RG. 4 IS a flow chart describing a method of the present environment of computer 102 as required to perform the 

invention as specifically adapted for white box testmg in ^^^^^^^^ t^st sequence. Invocation of program under test 104 

accordance with the present invention. executive 112 is indicated by arrow 154. Output gener- 

' DETAILED DESCRIPTION OF TOE ^^^^ ^7 Program under test 104 is capmred by output 

PREFERRED EMBODIMENTS analysis element 114 in test manager 108 as indicated by 

arrow 152. Output analysis 114 compares output values 

While the invention is susceptible to various modifica- generated by program under test 104 with expected values as 

tions and alternative forms, a specific embodiment thereof defined by the test sequence embedded within program 

has been shown by way of example in the drawings and will source code files 118. As used herein, output includes output 

herein be described in detail. It should be understood, values generated by the program under test 104 (as executed 

however, that it is not intended to limit the invention to the by the executive 112) as well as environmental side effects 

particular form disclosed, but on the contrary, the invention ^5 as discussed farther herein below. 

is to cover all modifications, equivalents, and alternatives FIG. 2 is a flow chart describing operation of methods of 

faUing within the spirit and scope of the invention as defined the present invention which automate program testing using 

by the appended claims. test sequences embedded within the programs source code. 

FIG. 1 is a block diagram of a system operable in The flow chart of FIG. 2 is representative of both black box 

accordance with the present invention to automate testing of 50 and white box testing techniques of the present invention, 

a computer program. Preferably, both program under test Differences between the two techniques are discussed fur- 

104 and automated test tool 100 are operable on computer ther herein below with regard to FIGS. 3 and 4. 

102^ Jtac polled Jn_ thc^art_wiU^readily_re^ Element 200 is first operable to parse source code files of 

automated test tool ,1 00_ and prograrn under 104 may v the program under test for purposes of locating a desired test 

equi^lently operate on different computers using well-'^ 55 identified by the user. Preferably, each test embedded within 

known distributed"computing"tecKniques^ ami cornmunica^ the source code files of the program under test is identified 

ction protoools. _ by a unique identifier (i.e., a name for the test). When the test 

AutomatedJest^tooLlW^incl^^ test sequencTextractor method of the present invention is invoked, the user pref- 

(106 operable to jsxtract test sequence information from] erably identifies the test by name which is to be performed, 
program source code files 118 stored on mass storage device- 60 In the preferred embodiment, a user may also request 
116 and connected to computer 102 via path 150. As above;^ execution of multiple named test or all tests embedded 

those skilled in the art will recognize that mass storage within the source code files of a program under test. In such 

device 116 may be locally attached to computer 102 or may a case, the method of FIG. 2 is repeated and element 200 is 

be accessed via remote distributed computing techniques operable to find a next desired test sequence embedded 

well-known to those skilled in the art. Specifically, test 65 within the program under test. 

sequence extractor 106 parses program source code files 118 Element 202 is next operable to determine whether the 

corresponding to program under test 104 for purposes of requested test (or next test) was successfully located in the 
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source code files of the program under test. If not, the Thai is, as shown in FIG. 2, element 204 completes its 

method of FIG. 2 is completed. Where the desired or next operation to extract a complete test sequence from source 

test is located by operation of elements 200 and 202, element code file before elements 206 through 210 operate to set up 

204 is next operable to extract the located test sequence from the environment, execute the test, and compare the output 
source code files of the program under test. In particular, as 5 results to expected results, respectively. Those skilled in the 

discussed further herein below with regard to the preferred art will recognize that elements 204 through 210 may 

syntax of test directives, element 204 extracts all hues operate in an iterative/incremental manner wherein their 

associated with the requested test as indicated by a marker operatioi^ are guided by the sequence of test directives and 

token (a unique string used to identify test directives within commands provided by the test user. In other words, each 
comment fields) lO ^^^^^^^^^ ^ interpreted and appropnate actions taken to 

J complete that directive before another directive is processed, 
nie test directives and parameters so extracted by opera- ^ ^^^^ ^.^^^^.^^ ^.^^^^^ ^^^^ 

tion of element 204 are then mterpreted by element 206 for ^^^^ j„ test sequence is to be performed relative 

purposes of settmg ttp any required environment of the ^ ^^^^^ ^^-^^ ^^^^ 

computing system for Operation of the test. For example, m j^,^ i a- a u- *u .u j fT-ir- 

performing a black box test as discussed further herein 15 FIG. 3 and is a flow chart descnbmg the method ^^^^^ 

below, particular environmental parameters of the comput- f detail with respect to operation for black box 

ing system as a whole may need to be initiaUzed (i.e., Jhell '".^'^8 echniques. As noted above black box testmg tech- 
variables, file system directory strucmres, etc.) Or, for "l^""^ °P"*"°" °^ ^ r'"^'"^ ^^^^1'°^ 

example, when performing a white box test as discussed ""P"^ ^'''f " '° -"^'onng corresponding 

further herein below, particular global variables within the 20 values to compare actual output values with e,q,ected 

, . * J * t_ • V- 1- J • . output values. For black box testmg purposes, the test 

program under test may need to be mitiahzed pnor to *^ . , , . J* ^"^^'^^^ 

t-ot directives and commands processed by the methods of the 
performing the test sequence. . »■ j e • * * 

° present mvention define input streams (files or other sources 

Element 208 is then operable to execute the test sequence ^^^^^^ ^o be applied to the program, and further define 

while captunng the generated output results therefrom. For expected output values to be generated by operation of the 

example, m performing a black box test sequence, element p.^gram in view of the provided test input streams. In 

208 IS operable to execute the program under test to comple- general, the input streams are specified in the test directives 
tion. Input streams required for performing the test were providing filenames for files containing desired test data, 

mitiahzed as required by element 206 above. Output stream manner, test directives and commands specify files 

capturing of the test results further comprises re-direction of containing expected output values for use in comparison 

the output streams generated by operation of the program ^g^j^^^ ^^^^^^ ^.p^^,^^ ^^^^^^ ^^^^^^ ^^t^^l ^^^^ ^ ^^^^^ 

under test. The output streams are redirected for capture into ^^e captured from the output generated by the program under 

a file and later comparison against expected results. ^est when executed and operating upon the provided test 

As noted above, captured output includes printouts gen- input streams, 
erated by the test executive (debugger tool) which reveal the Element 300 is first operable to parse source code files 

side effects on computer environmental variables. The test associated with the program under test and to extraa desired 

executive (debugger tool) is directed to display present test sequences. Element 300 is operable a manner analogous 

values of identified variables through the supported test to that of elements 200 through 204 described above with 

durectives of the test manager. These present values are j^spect to FIG. 2. Element 302 is then operable to set up 

captured along with other standard output generated by i^put streams for use by the program under test in accor- 

operation of the program for comparison with expected ^ance with test sequence extracted in operation of element 

300. As noted above, input streams arc preferably defined by 

Element 210 is then operable to compare the captured the test directives as files on a mass storage device associ- 

results (output values including any side effects of interest) ated with the computer. The files contain desired test input 
against expected results as specified in the test sequence 45 values to be used in verifying the operation of the program 

itself. For example, when performing a white box test under test. One or more such files will typically be provided 

sequence, desired values of local or global variables may be as required by the specifications of the program under test, 

comparedagainsttheacnialvaluesofsuch variables. In like For example, some programs may require but a single 

manner, return values from function invocation's may be stream of input while others may manipulate data extracted 
compared against expected values. jq from a plurality of input streams. 

Element 212 is then operable to determine whether the Element 304 is then operable to set up any required 

comparisons performed by element 210 are indicative of system environment parameters in accordance with param- 

success or failure of the test sequence. Where the compari- eters defined by the test sequence as extracted in element 

sons indicate successful test sequence operation, element 300. It is common in many computing systems to have 
214 is next operable to indicate successful completion of the 55 system parameters definable by a user. For example, Unix 

test sequence to the test user. Where comparisons by element shell or MS-DOS command processing programs both allow 

210 suggest failure of the test sequence, element 216 is for local values to be defined in a manner that is accessible 

operable to indicate to the user apparent failure of the test to programs run from the system. Such shell variables are 

sequence. In both cases, the method described by FIG. 2 is often required for operation of a program under test. A 
completed. 60 particular set of values associated with the shell variables 

Those skilled in the art will readily recognize that FIG. 2 may be desired or required to run the program under test for 

represents a generalization of both black box and white box purposes of testing a particular feaUire thereof. Element 304 

methods of the present invention for automating test proce- therefore is operable to set any such system environment 

dures. First, details regarding specific operations for black parameters required to perform the desired test sequence, 
box testing techniques and white box testing techniques are 65 Element 306 is then operable to redirect any output 

discussed in further detail herein below. Secondly, operation streams generated by execution of the program under test for 

of elements 204 through 210 are shown to be serial in nature. saving generated output in files on the mass storage system 
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associated with a computer. A program under lest may from invoked functions. These results of the test sequence 

generate one or more output streams in response to execu- are compared against expected values to determine success 

tion with particular input values provided. Each such output or failure of the test sequence. 

stream generated may be captured and saved in a fUe and for Element 400 is operable in like manner to that of element 

later comparison against expected values. 5 300 of FIG. 3 to parse source code files corresponding to the 

Element 308 is then operable to execute the dcsued program under test and extract a desired test sequence. As 

program and apply the specified input files to the appropnate ^^^^ .^ove, a user may specify a particular, single test 

mput streams for operation of the program. As noted above, ^ performed (identified by a name), or may 

output streams gene rated by the program s execution may be . w 1 u * * V c j u *u / 

*' , . ^ , .u J- ^ u 1 request multiple such tests be performed by the methods 

captured in accordance with the re-direction setup by ele- m a • i a • x^m a wtu w 1 u * + * a 

. lA-^r A J 1. . 1 ■ r i_i ^ ^ t-1 f i_ depicted in FIG. 4. Where multiple such tests are requested, 

ment 306. A debugger tool is preferably used for black box ,1, ,1. AT^m a * u a- * 1 * ^nn • li 

. . „ ™.. . T 1. . the method FIG. 4 repeats such that element 400 IS operable 

testmg as well as white box testing. In white box testms, as - * a - ^a * * u - j 

J ^jrL.LLi .1- j. t to obtam the next desired test sequence by parsmg and 

descnbed further below, the debugger tool IS used to invoke ^ * * r j ci r a. 

. , - . ... ^ . . . 1 extractmg the test from source code files of the program 

particular functions within a program and to set particular under te^ 

variables within the program for testing purposes. In black ' . 

box testing, the debugger tool may be used to force particu- Element 402 is next operable to construct the command 

lar error conditions within the program while executing to ^^"P^ P^?^^^^^ debugger for purposes of 

completion. This aUows for checking of proper operation of '^^^^^^^'^S ^^^^ sequence. In particular, the command 

the program, as a whole, in response to particular failure ^^"P^ P^«^^^^« commands to the debugger which: 

conditions within the program's execution. For example, a ^oals variables to specify values, 

common failure to detect is exhaustion of a particular Invoke desired procedures and functions with specific 

allocated resource used by the program (i.e., dynamic parameters, 

memory allocation or file storage allocation). In black box Compare results of function or procedure invocation with 

testing, the debugger tool is used to force a particular expected results, 

function call to fail to simulate such a resource allocation 25 Display present values of global variables, and 

failure condition. The test can then determine if the program Display function resuhs from invocation of functions in 

executes to completion with an appropriate "graceful" exit program under test 

or fails to perform as expected. -phese debugger commands of the command script are 
Upon completion of execution of the program under test, generated by operation of element 402 in accordance with 
element 310 is then operable to compare the captured output 30 the direaives of the test sequence extracted from the pro- 
streams with expected results in accordance with parameters gram sotu^ce code files. 

defined by the test sequence. In particular, the test sequence Element 404 is next operable to invoke the debugger 

defines files of expected output values to be used for program providing the generated command script as input 

comparison with the captured files generated by output thereto. The debugger program then executes the test 

streams of the program under lest. Lastly, element 312 is 35 sequence as represented by the debugger commands in the 

operable to return to the user an indication of success or command script file. The debugger, under direction of the 

failure in operation of the test sequence. generated command script, monitors and displays the 

Those skilled in the art will understand that FIG. 3 shows present value of global and local variables of interest to 

operation of elements 300 through 310 as serial or sequential detect proper operation of the program in accordance with 

in nature. It will be readily recognized that such operations 40 the lest sequence. In particular, the debugger tool is 

may be performed iteratively so as to repeatedly run one or instructed to display selected variable values and the test 

more programs while gathering generated output streams for executive is instructed to compare variable values or func- 

comparison against expected results. tion return values to expected values and notify the user of 

FIG. 4 is a flow chart describing additional details of the any errors in the program operation fi-om such comparisons, 

methods of the present invention as operable to perform 45 Output generated by the debugger while operating the 

white box testing techniques on a program under test. As program under test may be used for postprocessing analysis 

noted above, white box testing techniques of the present by an output analysis element (not shown). In the preferred 

invention are performed in conjunction with a program embodiment, the test executive is capable of comparing 

debugging tool (debugger) to permit precise control of the global variable values, local variable values, or function 

execution of particular modules or functions within the 50 return values, (displayed by the debugger tool) to expected 

program under test and to force particular errors in execution result values under direction of its command script in 

of a function or program. Those skilled in the art will accordance with test sequence. 

understand that a debugging tool as used herein means it tool Element 406 is finally operable to present to the user 

capable of performing incremental execution of particular information indicative of success or failure of the test 

procedures, functions, or modules within a program under 55 sequence. Success or failure is preferably indicated by 

test. Such debugging tools are well known to those skilled messages generated from the debugger program while oper- 

in the art and readily available for most commercial com- ating the program under test at the direction of the command 

puling systems. In general, white box testing techniques of script and underlying test sequence. Such messages may be 

the present invention permit directives embedded within generated to indicate that a particular global variable was not 

source code corresponding to the program under test to 60 set to the expected resulting value, or a function invocation 

request setting of global variables within modules of pro- returned an unexpected value, etc. 

gram under test, to request invocation of particular proce- Those skilled in the art will recognize a variety of 

dures or functions within program under test, and to evaluate commercially available debugger programs useful in white 

the results and side effects generated from operation of such box testing in accordance with the present invention, 

procedures or functions. Results and side effecls of such 65 Specifically, processing of element 404 in figure for may be 

operations are reflected in changes to global or local performed by a debugger known as "xdb" (readily available 

variables, generation of output streams, and return values in most Unix systems) or by a "gdb" (and variants thereof 
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readily available in most Unix systems through the Free 
Software Foundation). Processing of elements 402 and 404 
of FIG. 4 are therefore operable to generate appropriate 
commands in the command script as required by the par- 
ticular debugger selected. In the preferred embodiment, the 5 
"xdb" debugger is utilized for executing the command script 
built in accordance with test sequence and methods of the 
present invention. 

The methods of the present invention to perform black 
box testing or white box testing may be implemented using lo 
a Dumber of programming styles and languages. For 
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example, the test methods of the present invention may be 
implemented as custom program instructions in the C/C++ 
programming language. Or, for example, the methods may 
be implemented as shell scripts in a Unix or MS-DOS 
computing environment. Such design choices in implement- 
ing the techniques of the present invention are well known 
to those skilled in the art. 

The following shell script describes the preferred embodi- 
ment of the present invention for performing black box 
testing techniques. 



#! /bin/ksh -p 

Commando 

{ 

print -p "$1" 

print -p " p 1234S67S90" 

ResponseQ 
{ 

while read -p LINE; do 
LINE="${UNE#>}" 
LINE="${UNE#>}" 
LINE="${UNE#>}*' 
if [ "$UNE" - "1234567890" ]; then 
^ break 

fi 

echo "SLINE" 

done 

} 

if [ $# -tt 1 ]; then 

echo "Usage: ${0##*/} <program> [<testnum>]" 
exit 1 

fi 

rm -rf bb 
mkdir -p bb 
>bb/in 
nn -f core 

HOME=$PWD xdb -L -i bb/in -o bb/cut -e bb/err $1 2>&1 |& 

CoKimand "s" 

Response >/dev/null 

Command "z 13 sir" 

Response >/dcv/null 

Command "z 18 sir" 

Response >/dev/null 

Command "If 

FTLES-*Response | tail +2 | awk '{if ($2 I - "end.c") print $2}* [ grep -v'_' 
print -p *'k" 
print -p "y" 
Command 0" 
Re^nse >/dev/null 

egrep "*BLACKBOX:|*BOX:" $FILES | sed "sl'*BOX: >bb/lines 

TOTAL=0 

PASS=0 

S-0 

SED="s!@@!@!g" 
while read TEST, do 

if [ "${TEST##*@*}- - ]; then 

TEST='set I grep "TEST-. | sed "si 'TEST=II; $SED'" 

fi 

if [ "JTEST' != "SiTE^mmt}" ]; then 
((N-N+1)) 

if [ -2 "${2:-}" -o *'${2:-r = "$N" -o %{2:-y* - "0" J then 
if ["${2:-}" l«"0"];then 
echo 

fi 

echo "$N: $TEST" 

fi 

clif [ -z "$TEST" -o "$TEST' != «${TESW#}" J then 

if [ -2 "${2:-}" -o \( "${2:-}" = "$N" -a -$NP' t= "0" \) ]; then 
echo "$TEST' 

fi 

elif [ -z "${2:-}" -□ "$N" = "0" -o «${2:-}" = -$N" ]; then 
(CrOTAL+-l)) 
case ''${TBSr%% *}'* in 
exec:) 
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-continued 



EXEC--${TESr#excc:}" 
whUe [ -$EXEC' !- "SjEXECtf }" ]; do 
EXEC>."${EXEC# }" 

done 

if [ *'$EXEC' !- "${E?CEC%-=*}" ]; then 
ERVo"${EXEC##*— }" 
whUe [ "$ERV" !- "${ERV# }" ]; do 
ERV="${ERV# }" 

doDe 

EXEO"${EXEC%=-*}" 
else 

ERV-0 

ft 

if [ -n "SEXEC ]; then 
Command "r $EXEC' 

else 

Command "R" 

fi 

Responfie | grep " Child process " >bWexec 
if grcp "tcrnriinatcd normally" bb/cxcc >/dcv/nulI; then 
RV="(0)" 

elif RV-'grep **did an exit" bb/exec'; then 
RVo'echo "$RV" | awk '{ print $NF }" 

else 

RVo«(-l)" 

fi 

Command "db *" 

Response >/dcvynull 

tr -d "\015" <bb/out >bb/OUT 

mv bb/OUT bb/out 

tr -d "\015" <bb/err >bb/EEyi 

mv bb/ERR bb/crr 

>bb/iD 

exec 4<bb/out 5<ijb/err 
if C(RV!= ERV)); then 

echo "FAILED: $TEST" 

echo " EXPECTED: exit($ERV)" 

echo " RESPONSE: exitSRV" 

continue 

fi 

failoncc:|feil:) 

FAIL«"${TEST#fail:}" 
FAIL-"${FAIL#£ailonce:}" 
whUc [ "SFADL " !- "${FAIL# }" ]; do 
FAIL="${FAIL# }" 

done 

RV="${FAIL#* }" 

RV-'echo "$RV" | sed 's!;!; p !g" 

FAIL-"${FAIL%% *}" 

if [ •'${TEST%% •}" > "feilonce:" ]; then 

Command "ba $FA[L { Q ; bu { Q ; p \$r28 - $RV ; db * ; c } ; c }" 

else 

Command "ba $FA[L { Q ; bu { Q ; p \$r28 - $RV ; c } ; c }" 

fi 

Response >/dev/null 

file:)" 

FILE-«${TES™ie:}" 
while [ "SFILE" != "${FILE# }" ]; do 
FILE="${FILE# }" 

done 

rm -f $FILE 
touch $F1LE 

in:) " 

LINEo^'SlTESTWin:}" 
WhUe [ "SUNE" != "${LINE# }" ]; do 
LINE-"${LINE# }" 

done 

if [ "SLINE" to "<blackbox-eof>" ]; then 
echo "SLIME" »bb/Ln 

fi 

line:) 

LINE="${TESTWline: }" 
whUe [ "SLINE" != "${UNE# }" ]; do 
LINE=*'${LINE# }" 

done 
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echo "$UNE" »$FILE 
mode;) 

MODE="${TEST#mode:}" 
whUc [ "SMODE" != "${MODE# }" ]; do 
MODE-"${MODE# 

done 

chmod $MODE SFILE 
out:|cin) 

if [ "S{TEST%% •}•• = "out:" J then 
FD-4 

EXPECrEI>-"${TEST#out:}" 

else 

FDo5 

EXPECrED="${TEST#err:}" 

fi 

whUe [ "SEXPECTED" !» "${EXPECTEi:)# }" ]; do 
EXPECTED-"${EXPECTED# }" 

done 

if [ "SEXPECTED" = -<blac±box-eof>" ]; then 
read RESPONSE <&$FD 
if [ "$?" != 1 t then 

echo "FAILED: STEST' 
echo " EXPECTED: SEXPECTED" 
echo " RESPONSE: SRESPONSE" 
while read RESPONSE <&$FD; do 

echo " EXPECTED: SEXPECTED" 
echo " RESPONSE: SRESPONSE" 

done 
continue 

fi 

elif [ "SEXPECTED" !» ]; then 

read RESPONSE <&$FD 

if [ "$?" = 1 ]; then 

echo "FAILED: STEST' 

echo " EXPECTED: SEXPECTED" 

echo " RESPONSE: <blaclcbox-cof>" 

continue; 

else 

while [ "SRESPONSE" 1= "S{ RESPONSE* }" ]; do 
RESPONSE-"${RESPONSE# }" 

done 

if ( **${RESPONSE##$EXPECTED}" !- "" ]; then 
echo "FAILED: STESP' 
echo " EXPECTED: SEXPECTED" 
echo " RESPONSE: SRESPONSE" 
continue 

fi 

fi 

else 

while read RESPONSE <&$FD; do 
done 

fi 

shell;) 

SHELL="${TEST#sheli:}" 
while [ "SSHELL" !- -S{SHELL# }" ]; do 
SHELL="${SHELL# }" 

done 

cval SSHELL <bb/in >bb/out 2>bb/err 
>bb/in 

exec 4<bb/out 5<bb/err 
substi) 

SUBST="S(TEST#5ubst:}" 
whUe [ "SSUBST- t= -S{SUBST# }" I do 
SUBST-"${SUBST# }" 

done 

V="S{SUBST#* }" 
Vo'SV 

SUBST="${SUBST%% 
SED-*'s!@$SUBST!$V!g; $SED" 

•) 

echo -IGNORED: STESTT' 
continue 
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esac 

if [ "$ISr -gt 0 ]; then 

echo "passed: STEST' 

a 

((PASS+-1)) 

fi 

done <bb/lines 
print -p "q*' 
print -p "y" 
wait 

if [ -z "S{2:-}" -o "${2:-}" != "0" ]; then 

echo "$PASS of $TOTAL blackbox tests passed" 
if [ "SPASS" !- "STOTAL" ]; then 

((FAIL»TOTAL-PASS)) 

echo "$FAIL BLACKBOX TESTS FAILED" 

ft 

fi 

rm -rf bb 



The following shell script describes the preferred embodi- 
ment of the present invention for performing white box 
testing techniques. 



#! /bin/ksh -p 

Commando 

{ 

print -pr "$1" 

print -pr " p 1234567890" 

} 

RcsponscQ 
{ 

OXLINE- 

whilc :; do 

while read -p UNE; do 
UNE=*'${LINE#>}" 
UNE--${L[NE#>}" 
LINE=''${LINE#>}" 
if [ "SLINE' = "1234567890" J then 
break 

elif [ "SUNE*' != "${UP^#Cominand line or portion ignored:}" ]; then 
LINE-"${LINE#Command line or portion ignored:}" 
LINE-"${UNE#*\"}" 
LINEo"${UNE%\""}" 
XLINE="$XUNE ; $LINE" 
continue 

fi 

if [ -z "SOXLINE" ]; then 
echo "$LINE" 

a 

done 

if [ -n "SXLINE" J, then 
Command "SXLtNE" 
OXUNE=-$XLINE" 

else 

break 

a 

done 

} 

if [ $# -[t 1 ]; then 

echo "Usage: ${0##*/} «qDrogram> [<testnum>]" 
exit 1 

fi 

rm -rf wb 
mkdir -p wb 
rm -f core 

HOME-$PWD xdb -L$l 2>&1 |& 

Command "s" 

Response >/dev/null 

Command "va 0" 

Response >/dev/null 

Command "z 13 sir" 

Response >/dev/null 
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Command "z 18 sir'* 
Response >/dev/nult 
Command "p __IiutializcBFAO" 
Response >/dev/nuU 
Command "If 

FILES"' Response | tail +2 | awk '{if ($2 != "cnd.c") print $2}' | grep -v"_' 

egrep "*WHrrEBOX:l"BOX:" $FILES | ecd "s!* "BOX: *!!" >wb/Iinefi 

TOTALS 

FASS-0 

N=0 

SEDo*'s!@@!@!g" 
while read TESP, do 

if [ -SlTESW*©*}" = ); then 

TEST='set I grep 'TEST = | sed "st*TEST=n; $SED"' 

fi 

if [ -$TEST' != "${T1EST#####}" ]; then 
((N-N+D) 

if [ -z "${2:-}" -o "${2:-}" = "$N" -o "${2:-}" = "0" ]; then 
if I "${2:-}" I- -0" ]; then 
echo 

fi 

echo "$N: $TEST' 

fi 

continue 

clif [ -z *'$TEST" -o "STEST' !- "${TEST1###}" ]i then 

if I -z "${2:-}" -o \( "${2:-}" = -$N" -a "$isr !» "0" \) ]; then 
echo -$TEST' 

fi 

continue 

clif ( -z "${2:-}" -o "$ISP' = "0" -o "${2:-}" = "$N" I then 
((T0TAI>=1)) 
case "$[TEST%% *}" in 
failonce:|fail:) 

FAIU"${TEST#faU:}" 
FAII^"$ {FAIL#faaoncc: }" 
while [ -$FAIL " lo "${FAIL# }" ]; do 
FAIL-"${FAIL# }" 

done 

RV-"${FArL#* }" 

RV='ccho "$RV I sed 's!;!; p !g" 

FAIL="${FAIL%% 

if [ "${TEST%% - "fiiiloncc:" ]; then 

Command "ba $FAIL { Q ; bu {0 ; p \$r28 = $RV ; db * ; c } ; c }" 

else 

Command "ba $FAIL { Q ; bu { Q ; p \$r28 = $RV ; c } ; c }" 

fi 

Response >/dcv/null 
Kubst:) 

SUBSro"${TEST#5ubst:}" 
whUe [ *$SUBSr' 1= "${SUBST# }" }, do 
SUBST="${SUBST# }" 

done 

V-"${SUBST#* }" 
V='$V' 

SUBST="${SUBST%% 
SED-"8!@$SUBST!$V!g $SED" 

•) 

if [ -z «${TEST##'-»*}" ]; then 

COMMAND»"p ${TEST%%«»}" 
whUc [ "SCOMMAND" !» "${COMMAND% }" 1 do 
COMMAND="${O0MMAND% }" 

done 

RESULT="${TEST##*==}" 
while [ "SRESULT' 1= "$ {RESULT* }" ]; do 
RESUlT-"${REStlLT# }" 

done 

NAME-"${TE^%%[(]*}" 
NAME="${NAME##*[.>]}" 
if [ *'$RESULr' = "${ RESULTS 0-9<-]}-" I then 

Command "p SRESULT* 

RESUUr=*Response* 

RNAME-"${RESULT%%[C]*}" 

RNAME-"${RNAME##*[.>]}" 

RESULX-"${RESULT##$RNAME -}" 

while [ "SRESULT" != "${RESUL'I¥ }" ]; do 
RESULr=*'${RESULr# }" 

done 
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EXPECrED-"$NAME - $RESULT' 

else 

COMMAND-^p STEST' 
EXPECnED-"*" 

fi 

Command "JCOMMAND" 

RESPONSE-' Response' 

if [ "SEXFECTED" )- -a \ 

"${RESPONSE##$EXPECrEDV' !- ]; then 

print -r "FAILED: JTESTT' 

print -r " EXPECTED: SEXPECTED'* 

print -r " RESPONSE: SRESPONSE" 

continue 
e]if["$EXPECrED"-"*"-a\ 

"$RESPONSE" !- "${RESPONSE##* [(lUIjE}" ]; then 

print -r "FAILED: STEST' 

print -r " RESPONSE: SRESPONSE" 

continue 

fi 

Command "db *" 
Response >/dev/QulI 

esac 

if['W-gtO]-, then 

print -r "passed: JTESTT' 

fi 

(CPASS+.1)) 

fi 

done <wb/liiieB 

Command "p _UpdatcBFAO" 

Response >/dev/nuU 

print -p "q" 

print -p "y** 

wait 

if [ -2 "${2:-}" -o "${2:-}" != "0" ]; then 

echo "SPASS of STOTAL whitebox tests passed" 
if [ "$PASS" !- "STOTAL" ]; then 

((FAII^TOTAL-PASS)) 

echo "SFAIL WHITEBOX TESTS FAILED" 

fi 

fi 

rm -rf wb 



40 



The following test sequence directives (test commands) 
arc supported by the above discussed black box and white 
box shell scripts. Those skilled in the art will readily 
recognize a variety of additional test sequence directives 
which may be added to this list and supported by methods 
of the present invention. The following list of test directives ^5 
is therefore intended as exemplary and not as exclusive of 
other test directives. Further, those skilled in the art will 
readily recognize a variety of syntactic variations for 
expressing the test directives listed below. Such design 
choices for specifying test directives are well known to those 
in skilled in the art. As noted above, test directives are 
preferably embedded in source code files of the program 
under test within comments as defined by programming 
language of the source code files. 55 



Whitebox test directives include: 



Description 



Exemplary Syntax 



Declare new test 

Rule substitution (@name) 

Set global data 

Call procedures 



test ### 
Bubst: name command... 
globaldata - value 
procedure(args...) 



-continued 



Whitebox test directives include: 
Description Exemplary Syntax 



Call functions and verify return 
value 

Verify global data 



function (a rgs...) value 
globaldata == value 



50 



60 



65 



The following whitebox test example demonstrates use of 
some of the above directives embedded within a C language 
program. 



/* 

### globahnaxO ### 
global = 10 
globalmax(l, 2) 10 
globatmax(20, 2) -» 20 
globalmaxfl, 30) « 30 
-/ 

inl global; 
globalmax(al, a2) { 

if (al >= a2 && al >= global) return al; 

if (a2 >= al && a2 >= global) return a2; 

return global; 

} 
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Description 



Blackbox test directives include: 
Exemplary Syntax 



Start new test 


iHHt test ffitU 


Rule Substitution (@name) 


subst: name command ... 


Create file 


file: name 




mode: mode 




line: line 




line: <eof> 


Set up stdin for next 


in: line 


"exec" 01 "shell" 






in: <eof> 


Force function to fail 


(ail: function value 


at next "exec" 




Exec program (<in >out 2>err) 


exec: args ... 


Exec shell command 


shell: command ... 


(<in >out 2>err) 




Verify stdout of last 


out: line (may contain shell wildcards) 


"exec" or "shell" 






out: <eof> 


Verify stderr of last 


err: line (may contain shell wildcards) 


"exec" or "shell" 






err: <cof> 



10 



The following blackbox test example demonstrates use of 
some of the above directives embedded within a C language 
program. 



fid* 



subst: AMI whoami 

exec: -un 

out: @AMI 

out: <eofo- 

err: <eDf> 

fail: getuid 0 

exec: -un 

out: root 

out: <eofc- 

err: <eof> 



30 



35 



40 



While the invention has been illustrated and described in 
detail in the drawings and foregoing description, such illus- 
tration and description is to be considered as exemplary and 45 
not restrictive in character, it being understood that only the 
preferred embodiment and minor variants thereof have been 
shown and described and that all changes and modifications 
that come within the spirit of the invention are desired to be 
protected. 50 

What is claimed is: 

1. A system for testing a computer program comprising: 

a test sequence extractor which extracts a test sequence 
from a source code file corresponding to said program 55 
wherein said test sequence is embedded in said source 
code file in a manner transparent to the underlying 
programming language within said source code file; 
and 

a test manager, coupled to said test sequence extractor, 60 
which performs said test sequence by applying input 
information associated with said test sequence to said 
program and evaluating output information generated 
by said program to determine whether said program is 
operating properly. 65 

2. The system of claim 1 wherein said test sequence 
comprises comments embedded within said source code file. 



3. The system of claim 1 wherein said test manager 
includes: 

a test sequence interpreter which interprets said test 
sequence; 

a test executive, responsive to said test sequence 
interpreter, which invokes said program in accordance 
with said test sequence and which applies said input 
information to said program in accordance with said 
test sequence; and 

a test output processor, responsive to said test sequence 
interpreter, which evaluates said output information in 
accordance with said test sequence. 

4. The system of claim 3 wherein said test output pro- 
cessor includes: 

an output capture element which receives said output 
information from said program; and 

an output comparator element which compares the cap- 
tured output information with expected output infor- 
mation in accordance with said test sequence. 

5. The system of claim 3 wherein said test executive 
comprises: 

a program debugging tool. 

6. The system of claim 5 wherein said program debugging 
tool includes: 

a subfunction invocation element which invokes an iden- 
tified subfunction within said program in accordance 
with said test sequence. 

7. A software test tool for performing white box testing of 
a program, said test tool comprising: 

a test sequence interpreter which interprets a predefined 
test sequence wherein said predefined test sequence is 
embedded in a source code file corresponding to said 
program in a manner transparent to the underlying 
programming language within said source code file; 
and 

a test sequence executive, coupled to said test sequence 
interpreter, which invokes an identified subfunction 
within said program in accordance with said test 
sequence to verify proper operation of said subfunc- 
tion. 

8. The test tool of claim 7 wherein said test sequence 
interpreter includes: 

a test sequence extractor which extracts said test sequence 
from said source code file associated with said pro- 
gram. 

9. The test tool of claim 8 wherein said test sequence is 
encoded as a comment in the source code file. 

10. The test tool of claim 7 wherein said test sequence 
executive comprises a debugger. 

11. The test tool of claim 10 wherein said debugger 
includes: 

means for applying input information associated with said 
test sequence to said identified subfunction; and 

means for capturing output information generated by said 
identified subfunction. 

12. The test tool of claim 11 wherein said test executive 
further comprises: 

means for comparing the captured output information 
with expected results associated with said test sequence 
to verify proper operation of said identified subfunc- 
tion. 

13. A method for testing a computer program comprising 
the steps of: 

extracting a test sequence from a source code file corre- 
sponding to said program wherein said test sequence is 
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embedded in said source code file in a manner trans- 
parent to the underlying programming language within 
said source code file; 
executing said test sequence by applying input informa- 
tion associated with said test sequence to said program; ^ 
and 

evaluating output information generated by said program 
to determine whether said program is operating prop- 
erly. 

14. The method of claim 13 wherein said test sequence 
comprises comments embedded within said source code file. 

15. The method of claim 13 wherein the step of executing 
includes the steps of: 

interpreting said test sequence; 

invoking said program in accordance with said test 
sequence; 

applying said input information to said program in accor- 
dance with said test sequence; and 

evaluating said output information in accordance with 
said test sequence. 

16. The method of claim 15 wherein said the step of 
evaluating includes the steps of: 

capturing said output information from said program; and 
comparing the captured output information with expected 

output information in accordance with said test 

sequence. 

17. The method of claim 15 wherein the step of executing 
is performed by a program debugging tool. 

18. The method of claim 17 wherein the step of invoking 
includes the step of: 

invoking an identified subfunction within said program in 
accordance with said test sequence. 
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19. A method for performing white box testing of a 
program, said method comprising the steps of: 

interpreting a predefined test sequence wherein said pre- 
defined test sequence is embedded in a source code file 
corresponding to said program in a manner transparent 
to the underlying programming language within said 
source code file; and 

invoking an identified subfunction within said program in 
accordance with said test sequence to verify proper 
operation of said subfunction. 

20. The method of claim 19 wherein the step of inter- 
preting includes the step of: 

extracting said test sequence from said source code file 
associated with said program. 

21. The method of claim 20 wherein said test sequence is 
encoded as a comment in the source code file. 

22. The method of claim 19 wherein the step of invoking 
is performed by a debugger. 

23. The method of claim 22 wherein the step of invoking 
includes the steps of: 

applying input information associated with said test 
sequence to said identified subfunction; and 

capturing output information generated by said identified 
subfunction. 

24. The method of claim 23 wherein the step of executing 
further comprises the step of: 

comparing the captured output information with expected 
results associated with said test sequence to verify 
proper operation of said identified subfunction. 
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